When does Dedicated Protocol Processing Make Sense?

Distributed-memory parallel computers and networks of workstations (NOWs) both rely on efficient communication over increasingly high-speed networks. Software communication protocols— from flow-control and reliable delivery to multicasting and coherent distributed shared memory— are often the performance bottleneck. Several current and proposed parallel systems—e.g., the Intel Paragon—address this problem by dedicating one general-purpose processor (in a multiprocessor node) specifically for protocol processing. This operating system convention reduces communication latency and increases effective bandwidth, but also reduces the peak performance since the dedicated processor no longer performs ‘‘useful’’ computation. In this paper, we study a network of multiprocessor workstations and ask the question: ‘‘when does it make sense to dedicate a processor specifically for protocol processing?” We compare three protocol processing policies: Single, the baseline case with one processor that does everything; Fixed, which uses a dedicated protocol processor; and Floating, where all processors perform both computation and protocol processing. We use a simple analytic model of a general request/reply protocol to illustrate the trade-offs between the policies. The model shows that: i) adding a dedicated protocol processor to a uniprocessor node is unlikely to be cost-effective and even less likely to outperform the Floating policy; ii) a dedicated processor is more advantageous for light-weight protocols (e.g., active messages) than for heavy-weight protocols (e.g., TCP/IP), iii) the Fixed policy becomes advantageous when communication becomes the bottleneck, as when multiple compute processors and multithreading saturate the resource. The break-even point between Fixed and Floating is a function of the number of processors, protocol overheads, and application parallelism. We then evaluate these policies in the context of a fine-grain user-level distributed shared memory system. We present preliminary measurements from a dedicated network of Sun SparcStation-20s connected by a Myrinet network. The measured performance on four nodes—each with up to four processors—of three hybrid shared-memory parallel applications confirm the intuitive results from the model.